{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EIdT9iu_Z4Rb"
   },
   "source": [
    "# Training a classification model for wine production quality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective:\n",
    "\n",
    "In this lab, you will use the Keras Sequential API to create a classification model. You will learn how to use the tf.data API for creating input pipelines and use feature columns to prepare the data to be consumed by a neural network. \n",
    "\n",
    "### Lab Scope: \n",
    "This lab does not cover how to make predictions on the model or deploy it to Cloud AI Platform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning objectives:\n",
    "1. Apply techniques to clean and inspect data.\n",
    "2. Split dataset into training, validation and test datasets.\n",
    "3. Use the tf.data.Dataset to create an input pipeline.\n",
    "4. Use feature columns to prepare the data to be tained by a neural network.\n",
    "5. Define, compile and train a model using the Keras Sequential API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AHp3M9ZmrIxj"
   },
   "source": [
    "In a *classification* problem, we aim to select the output from a limited set of discrete values, like a category or a class. Contrast this with a *regression* problem, where we aim to predict a value from a continuos range of values.\n",
    "\n",
    "This notebook uses the [Wine Production Quality](https://archive.ics.uci.edu/ml/datasets/wine+quality) Dataset and builds a model to predict the production quality of wine given a set of attributes such as its citric acidity, density, and others. \n",
    "\n",
    "To do this, we'll provide the model with examples of different wines produced, that received a rating from an evaluator. The ratings are provided by the numbers 0 - 10 (0 being of very poor quality and 10 being of great quality).  We will then try and use this model to predict the rate a new wine will receive by infering towards the trained model.\n",
    "\n",
    "Since we are learning how to use the Tensorflow 2.x API, this example uses the `tf.keras` API. Please see [this guide](https://www.tensorflow.org/guide/keras) for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "moB4tpEHxKB3"
   },
   "outputs": [],
   "source": [
    "# Ensure the right version of Tensorflow is installed.\n",
    "!pip freeze | grep tensorflow==2.1 || pip install tensorflow==2.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1rRo8oNqZ-Rj"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import pathlib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F_72b0LCNbjx"
   },
   "source": [
    "## The Wine Quality Dataset\n",
    "\n",
    "The dataset is available in the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gFh9ne3FZ-On"
   },
   "source": [
    "### Get the data\n",
    "There is a copy of the White Wine dataset available on Google Cloud Storage (GCS). \n",
    "The cell below shows the location of the CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p9kxxgzvzlyz"
   },
   "outputs": [],
   "source": [
    "dataset_path = \"gs://cloud-training-demos/wine_quality/winequality-white.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nslsRLh7Zss4"
   },
   "source": [
    "To visualize and manipulate the data, we will use [pandas](https://pandas.pydata.org/docs/).\n",
    "\n",
    "First step is to import the data. We should list the columns that will be used to train our model. These column names will define what data will compose the [dataframe](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) object in pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names = ['fixed_acidity','volatile_acidity','citric_acid','residual_sugar',\n",
    "                'chlorides','free_sulfur_dioxide','total_sulfur_dioxide','density',\n",
    "                'pH','sulphates','alcohol','quality']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_dataframe = pd.read_csv(dataset_path, names=column_names, header = 0, \n",
    "                      na_values = \" \", comment='\\t',\n",
    "                      sep=\";\", skipinitialspace=True)\n",
    "\n",
    "raw_dataframe = raw_dataframe.astype(float)\n",
    "raw_dataframe['quality'] = raw_dataframe['quality'].astype(int)\n",
    "dataframe= raw_dataframe.copy()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean the data\n",
    "\n",
    "Datasets sometimes can have null values. Running the next cell counts how many null values exist on each one of the columns. \n",
    "\n",
    "**Note**: There are many other steps to make sure the data is clean, but this is out of the scope of this exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframe.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that, on this dataset, there are no null values.\n",
    "If there were any, we could run `dataframe = dataframe.dropna()` to drop them and make this tutorial simpler."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect the data\n",
    "\n",
    "Let's take a look at the dataframe content. The tail() method, when ran on a dataframe, shows the last n roles (n is 5 by default)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframe.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_stats = dataframe.describe()\n",
    "data_stats = data_stats.transpose()\n",
    "data_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a quick look at the joint distribution of a few pairs of columns from the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "sns.pairplot(dataframe[[\"quality\", \"citric_acid\", \"residual_sugar\", \"alcohol\"]], diag_kind=\"kde\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### --- Some considerations ---\n",
    "\n",
    "Did you notice anything when looking at the stats table?\n",
    "\n",
    "One useful piece of information we can get from those are, for example, min and max values. This allows us to understand ranges in which these features fall in.\n",
    "\n",
    "Based on the description of the dataset and the task we are trying to achieve, do you see any issues with the examples we have available to train on?\n",
    "\n",
    "**Did you notice** that the ratings on the dataset range from 3 to 9? In this dataset, there is no wine rated with a 10 or a 0 - 2 rating. This will likely produce a poor model that is not able to generalize well to examples of fantastic tasting wine (nor to the ones that taste pretty bad!).\n",
    "One way to fix this is to make sure your dataset represents all possible classes well. Another analysis, that we do not do on this exercise, is check if the data is balanced. Having a balanced dataset produces fair model, and that is always a good thing!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data into train, validation and test\n",
    "\n",
    "Now split the dataset into a training, validation, and test set.\n",
    "\n",
    "Test sets are used for a final evaluation of the trained model.\n",
    "\n",
    "There are more sophisticated ways to make sure that your splitting methods are repeatable. Ideally, the sets would always be the same after splitting to avoid randomic results, which makes experimentation difficult."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(dataframe, test_size=0.2)\n",
    "train, val = train_test_split(train, test_size=0.2)\n",
    "print(len(train), 'train examples')\n",
    "print(len(val), 'validation examples')\n",
    "print(len(test), 'test examples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the tf.data.Dataset\n",
    "\n",
    "The tf.data.Dataset allows for writing descriptive and efficient input pipelines. Dataset usage follows a common pattern:\n",
    "\n",
    "- Create a source dataset from your input data.\n",
    "- Apply dataset transformations to preprocess the data.\n",
    "- Iterate over the dataset and process the elements.\n",
    "\n",
    "Iteration happens in a streaming fashion, so the full dataset does not need to fit into memory.\n",
    "\n",
    "The `df_to_dataset` method below creates a dataset object from a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def df_to_dataset(dataframe, epochs=10, shuffle=True, batch_size=64):\n",
    "  dataframe = dataframe.copy()\n",
    "  labels = tf.keras.utils.to_categorical(dataframe.pop('quality'), num_classes=11) #extracting the column which contains the training label\n",
    "  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))\n",
    "  if shuffle:\n",
    "    ds = ds.shuffle(buffer_size=len(dataframe))\n",
    "  ds = ds.repeat(epochs).batch(batch_size)\n",
    "  return ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to create batches from train, validation and test datasets that we split earlier. Let's use a batch size of 5 for demonstration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ds = df_to_dataset(train)\n",
    "val_ds = df_to_dataset(val, shuffle=False)\n",
    "test_ds = df_to_dataset(test, shuffle=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at one batch of the data. The example below prints the content of a batch (column names, elements from the `citric_acid` column and elements from the `quality` label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for feature_batch, label_batch in train_ds.take(1):\n",
    "  print('Every feature:', list(feature_batch.keys()))\n",
    "  print('A batch of citric acid:', feature_batch['citric_acid'])\n",
    "  print('A batch of quality:', label_batch )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create feature columns\n",
    "\n",
    "TensorFlow provides many types of feature columns. In this exercise, all the feature columns are of type `numeric`. If there were any text or categorical values, transformations would need to take place to make the input all numeric.\n",
    "\n",
    "However, you often don't want to feed a number directly into the model, but instead split its value into different categories based on numerical ranges. To do this, use the `bucketized_column` method of feature columns. This allows for the network to represent discretized dense input bucketed by boundaries.\n",
    "\n",
    "Feature columns are the object type used to create `feature layers`, which we will feed to the Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import feature_column\n",
    "\n",
    "feature_columns = []\n",
    "\n",
    "fixed_acidity = tf.feature_column.numeric_column('fixed_acidity')\n",
    "bucketized_fixed_acidity = tf.feature_column.bucketized_column(\n",
    "    fixed_acidity, boundaries=[3., 5., 7., 9., 11., 13., 14.])\n",
    "feature_columns.append(bucketized_fixed_acidity)\n",
    "\n",
    "volatile_acidity = tf.feature_column.numeric_column('volatile_acidity')\n",
    "bucketized_volatile_acidity = tf.feature_column.bucketized_column(\n",
    "    volatile_acidity, boundaries=[0., 0.2, 0.4, 0.6, 0.8, 1.])\n",
    "feature_columns.append(bucketized_volatile_acidity)\n",
    "\n",
    "citric_acid = tf.feature_column.numeric_column('citric_acid')\n",
    "bucketized_citric_acid = tf.feature_column.bucketized_column(\n",
    "    citric_acid, boundaries=[0., 0.4, 0.7, 1.0, 1.3, 1.8])\n",
    "feature_columns.append(bucketized_citric_acid)\n",
    "\n",
    "residual_sugar = tf.feature_column.numeric_column('residual_sugar')\n",
    "bucketized_residual_sugar = tf.feature_column.bucketized_column(\n",
    "    residual_sugar, boundaries=[0.6, 10., 20., 30., 40., 50., 60., 70.])\n",
    "feature_columns.append(bucketized_residual_sugar)\n",
    "\n",
    "chlorides = tf.feature_column.numeric_column('chlorides')\n",
    "bucketized_chlorides = tf.feature_column.bucketized_column(\n",
    "    chlorides, boundaries=[0., 0.1, 0.2, 0.3, 0.4])\n",
    "feature_columns.append(bucketized_chlorides)\n",
    "\n",
    "free_sulfur_dioxide = tf.feature_column.numeric_column('free_sulfur_dioxide')\n",
    "bucketized_free_sulfur_dioxide = tf.feature_column.bucketized_column(\n",
    "    free_sulfur_dioxide, boundaries=[1., 50., 100., 150., 200., 250., 300.])\n",
    "feature_columns.append(bucketized_free_sulfur_dioxide)\n",
    "\n",
    "total_sulfur_dioxide = tf.feature_column.numeric_column('total_sulfur_dioxide')\n",
    "bucketized_total_sulfur_dioxide = tf.feature_column.bucketized_column(\n",
    "    total_sulfur_dioxide, boundaries=[9., 100., 200., 300., 400., 500.])\n",
    "feature_columns.append(bucketized_total_sulfur_dioxide)\n",
    "\n",
    "density = tf.feature_column.numeric_column('density')\n",
    "bucketized_density = tf.feature_column.bucketized_column(\n",
    "    density, boundaries=[0.9, 1.0, 1.1])\n",
    "feature_columns.append(bucketized_density)\n",
    "\n",
    "pH = tf.feature_column.numeric_column('pH')\n",
    "bucketized_pH = tf.feature_column.bucketized_column(\n",
    "    pH, boundaries=[2., 3., 4.])\n",
    "feature_columns.append(bucketized_pH)\n",
    "\n",
    "sulphates = tf.feature_column.numeric_column('sulphates')\n",
    "bucketized_sulphates = tf.feature_column.bucketized_column(\n",
    "    sulphates, boundaries=[0.2, 0.4, 0.7, 1.0, 1.1])\n",
    "feature_columns.append(bucketized_sulphates)\n",
    "\n",
    "alcohol = tf.feature_column.numeric_column('alcohol')\n",
    "bucketized_alcohol = tf.feature_column.bucketized_column(\n",
    "    alcohol, boundaries=[8., 9., 10., 11., 12., 13., 14.])\n",
    "feature_columns.append(bucketized_alcohol)\n",
    "\n",
    "\n",
    "feature_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a feature layer from the feature columns\n",
    "\n",
    "feature_layer = tf.keras.layers.DenseFeatures(feature_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define, compile and train the Keras model\n",
    "\n",
    "We will be using the [Keras Sequential API](https://keras.io/api/models/sequential/) to create the logistic regression model for the classification of the wine quality.\n",
    "\n",
    "The model will be composed of the input layer (feature_layer created above), a single dense layer with two neural nodes, and the output layer, which will allow the model to predict the rating (1 - 10) of each instance being inferred.\n",
    "\n",
    "When compiling the model, we define a loss function, an optimizer and which metrics to use to evaluate the model. `CategoricalCrossentropy` is a type of loss used in classification tasks. Losses are a mathematical way of measuring how wrong the model predictions are.\n",
    "\n",
    "Optimizers tie together the loss function and model parameters by updating the model in response to the output of the loss function. In simpler terms, optimizers shape and mold your model into its most accurate possible form by playing with the weights. The loss function is the guide to the terrain, telling the optimizer when it’s moving in the right or wrong direction. We will use `Adam` as our optimizer for this exercise. Adam is an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data.\n",
    "\n",
    "There are many types of optimizers one can chose from. Ideally, when creating an ML model, try and identify an optimizer that has been empirically adopted on similar tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "  feature_layer,\n",
    "  layers.Dense(8, activation='relu'),\n",
    "  layers.Dense(8, activation='relu'),\n",
    "  layers.Dense(11, activation='softmax')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=False),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(train_ds,\n",
    "          validation_data=val_ds,\n",
    "          epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When training a model, you want to evaluate its performance by looking at the loss and the chosen metric(s).\n",
    "The validation loss and accuracy will point out if the model is actually learning and able to generalize or if it is overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vgGQuV-yqYZH"
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook introduced a few concepts to handle a classification problem with Keras Sequential API.\n",
    "\n",
    "* We looked at some techniques to clean and inspect data.\n",
    "* We split the dataset into training, validation and test datasets.\n",
    "* We used the tf.data.Dataset to create an input pipeline.\n",
    "* We went over some basics on loss and optimizers.\n",
    "* We covered the steps to define, compile and train a model using the Keras Sequential API."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "regression.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
